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Preface  &  Acknowledgements 


Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  &  Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SFIIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
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•  Office  of  Procurement  and  Assistance  Management  Headquarters,  Department 
of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test  & 
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•  Program  Executive  Officer,  Tactical  Aircraft 
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Mark  Kryzsko — Mr.  Krzysko  serves  as  the  deputy  director  of  the  Enterprise  Information  and  Office  of 
the  Secretary  of  Defense  Studies.  In  this  senior  leadership  position,  he  oversees  Federally  Funded 
Research  and  Development  Centers  and  directs  data  governance,  technical  transformation,  and 
shared  services  efforts  to  make  timely,  authoritative  acquisition  information  available  to  support 
oversight  of  the  Department  of  Defense’s  major  programs — a  portfolio  totaling  more  than  $1.6  trillion 
of  investment  funds  over  the  life  cycle  of  the  programs. 

Preceding  his  current  position,  Mr.  Krzysko  served  as  ADUSD  for  business  transformation, 
providing  strategic  guidance  for  re-engineering  the  Department’s  business  system  investment 
decision-making  processes.  He  also  served  as  ADUSD  for  strategic  sourcing  &  acquisition  processes 
and  as  director  of  the  Supply  Chain  Systems  Transformation  Directorate,  championing  and  facilitating 
innovative  uses  of  information  technologies  to  improve  and  streamline  the  supply  chain  process  for 
the  Department  of  Defense.  As  the  focal  point  for  supply  chain  systems,  he  was  responsible  for 
transformation,  implementation,  and  oversight  of  enterprise  capabilities  for  the  acquisition,  logistics, 
and  procurement  communities.  In  addition,  Mr.  Krzysko  served  as  advisor  to  the  deputy  under 
secretary  of  defense  for  business  transformation  on  supply  chain  matters  and  as  the  functional 
process  proponent  to  the  Department’s  business  transformation  efforts,  resulting  in  the  establishment 
of  the  Business  Transformation  Agency. 

In  March  2002,  Mr.  Krzysko  joined  the  Defense  Procurement  and  Acquisition  Policy  office  as 
deputy  director  of  e-business.  As  the  focal  point  for  the  acquisition  domain,  he  was  responsible  for 
oversight  and  transformation  of  the  acquisition  community  into  a  strategic  business  enterprise.  This 
included  driving  the  adoption  of  e-business  practices  across  the  Department,  leading  the  move  to 
modernize  processes  and  systems,  and  managing  the  investment  review  process  and  portfolio  of 
business  systems.  Mr.  Krzysko  served  as  the  division  director  of  Electronic  Commerce  Solutions  for 
the  Naval  Air  Systems  Command  from  June  2000  to  March  2002.  From  April  1991  until  March  2000, 
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Mr.  Krzysko  served  in  various  senior-level  acquisition  positions  at  the  Naval  Air  Systems  Command, 
including  contracting  officer  of  F/A-1 8  foreign  military  sales,  F/A-18  developmental  programs,  and  the 
F-14.  In  addition,  he  served  as  program  manager  of  Partnering,  the  Acquisition  Business  Process  Re¬ 
engineering  Effort,  and  as  acquisition  program  manager  for  the  Program  Executive  Office  for  Tactical 
Aircraft. 

Mr.  Krzysko  began  his  career  in  the  private  sector  in  various  executive  and  managerial  positions, 
including  assistant  managing  director  for  Lord  &  Taylor  Department  Stores  and  operations 
administrator  for  Woodward  &  Lothrop  Department  Stores.  Mr.  Krzysko  holds  a  Bachelor  of  Science 
degree  in  finance  from  the  University  of  Maryland  University  College,  College  Park,  MD,  and  a  Master 
of  General  Administration  degree  in  financial  management  from  the  same  institution. 
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of  Massachusetts  at  Amherst  in  1998  and  2003,  respectively.  Professor  Raja’s  research  focus  is  in 
the  field  of  artificial  intelligence,  specifically  as  it  relates  to  the  study  of  decentralized  control  and 
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Carolina  at  Charlotte,  a  senior  fellow  at  the  Center  for  Excellence  in  Municipal  Management  at 
George  Washington  University;  and  a  visiting  scientist  at  Software  Engineering  Institute  at  Carnegie 
Mellon  University  (2007-2008).  Dr.  Brown  has  extensive  experience  in  cross-organizational 
information  systems  integration  in  government  and  in  researching  the  development  and  design  of  a 
program  methodology  for  the  acquisition  of  joint  information  systems.  Her  research  interests  center 
on  participatory  design,  knowledge  management,  and  joint  problem  solving  and  program  planning. 
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Abstract 

Our  research  goal  is  to  proactively  model  the  non-linear  cascading  effects  of 
interdependencies  in  highly  dependent  networks.  Specifically,  we  examine  Department  of 
Defense  (DoD)  acquisition  from  the  context  of  the  joint  space  of  Major  Defense  Acquisition 
Programs  (MDAPs),  the  space  where  MDAPs  exchange  and  share  resources  for  the  purpose 
of  establishing  joint  capabilities.  Our  hypothesis  is  that  examining  the  interdependent  regions 
among  MDAPs  from  multiple  perspectives  using  non-linear  methods  will  allow  for  “what-if 
analyses  and  will  help  decision-makers  gain  insight  into  the  cascading  effects  of 
perturbations  and  take  appropriate  measures  to  handle  them.  Additionally,  we  also  ascertain 
whether  a  popular  decision  theoretic  model  for  decision-making  and  planning  for  cascading 
effects  in  the  face  of  uncertainty  is  appropriate  to  study  the  cascading  effects  among  MDAPs. 
Our  approach  is  to  use  a  case  study  to  determine  whether  the  data  required  to  build  an 
effective  decision-theoretic  model  is  available.  We  also  capture  the  data  investigation  process 
and  identify  the  challenges  that  were  encountered.  Our  results  show  that  it  is  possible  to 
recast  the  study  of  cascading  effects  in  MDAPs  as  a  sequential  decision  problem.  We  also 
have  captured  the  informational  value  in  the  existing  data  and  the  challenges  inherent  in  the 
data  collection  process. 

Introduction 

In  this  research,  we  seek  to  understand  and  model  the  behavior  of  non-linear 
cascading  effects  in  the  joint  space  of  Major  Defense  Acquisition  Programs  (MDAPs)  where 
their  transactions  form  interdependencies.  The  flows  in  and  out  of  an  MDAP  can  be 
examined  to  conduct  scenario  planning  or  “what-if  analyses.  These  “what-if  analyses  will 
help  decision-makers  gain  insight  into  the  cascading  effects  of  perturbations  and  take 
appropriate  measures  to  handle  them.  We  have  developed  models  that  can  address  “what- 
if  scenarios:  What  if  my  partner  reneges  on  a  funding  obligation?  What  if  Congress  alters 
my  funding?  How  will  the  perturbation  affect  my  partners?  We  also  identify  and  enumerate 
the  characteristics  in  the  existing  MDAP  data  that  are  critical  to  building  a  complete  model  of 
MDAP  behavior  and  discuss  the  challenges  in  acquiring  some  of  this  data  so  that 
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appropriate  governance  mechanisms  can  then  be  isolated.  We  emphasize  this  data 
acquisition  process  as  much  as  the  behavioral  findings  with  the  hope  that  the  lessons 
learned  from  the  process  would  allow  for  more  accurate  and  complete  data  gathering  and 
modeling  in  future  iterations  of  this  work. 

The  MDAP  data  that  we  analyzed  included  selected  acquisition  reports  (SARs), 
defense  acquisition  executive  summaries  (DAESs)  and  program  element  (PE)  documents 
over  multiple  years.  Although  our  aim  was  to  work  on  the  entire  collection  of  MDAPs,  we 
observed  that  this  eclectic  conglomeration  of  information  is  highly  unstructured,  significantly 
inordinate,  and  unmanageably  colossal  for  manual  analyses.  Hence,  we  focused  on  a  case 
study  that  contains  a  small  set  of  existing  MDAPs.  We  used  fictitious  names  (e.g.,  MDAP_A, 
MDAPJ3,  etc.)  to  retain  confidentiality  of  individual  program  information.  In  this  case  study, 
we  did  an  in-depth  analysis  of  the  data  and  studied  their  complex  interrelationships  from 
multi-perspectives  with  the  hope  that  some  of  our  observations  and  lessons  learned  about 
MDAPs  and  the  analysis  process  can  then  be  scaled  to  the  entire  network. 

Background 

The  decade-old  joint  capabilities  paradigm  at  the  Department  of  Defense  (DoD)  aims 
to  achieve  interagency  cooperation.  Vice  Chairman  of  the  Joint  Chiefs  of  Staff  Admiral 
Giambastiani  (2004)  claimed  that  the  integrated  force  had  to  become  interdependent  by 
being  capabilities-based,  collaborative,  and  network  centric.  This  collaborative  approach 
necessitates  integration  of  three  distinct  processes,  such  as  the  congressional  budgeting 
justification  process,  the  acquisition  process,  and  the  system  requirements.  It  is  observed, 
however,  that  the  acquisition  process  has  been  largely  tailored  to  suit  the  needs  of  the 
distinct  discrete  programs  without  addressing  the  interdependency  issues.  To  be  specific, 
although  many  MDAPs  are  entitled  to  joint  status  according  to  their  SARs,  DAES  reports 
and  milestone  reviews  tend  to  evaluate  the  program  performance  from  an  individual 
program  point  of  view,  irrespective  of  the  joint  space.  There  is  reason  to  believe  that  the 
exogenous  issues  generated  from  the  shared  domains  remain  unnoticed  to  the  extent  of 
causing  the  program  to  potentially  experience  severe  performance  degradation  (Brown, 
2011). 

Although  it  is  critically  important  to  understand  the  program  interfaces  and 
interdependencies,  there  are  few  tested  and  proven  tools  for  program  managers  and 
acquisition  executives  to  probe  the  joint  space  or  to  track  the  cascading  effects  that  the  joint 
space  might  trigger.  We  harnessed  a  network-centric  approach  to  study  DoD  acquisition  and 
focused  on  an  MDAP  network  of  interrelated  programs  that  exchange  and  share  resources 
for  the  purpose  of  establishing  joint  capabilities. 

We  studied  whether  performance  breaches  correlate  with  interdependency 
characteristics  in  the  context  of  the  JTRS  network.  We  also  studied  how  the  various  models 
can  be  used  to  determine  what  elements  of  the  models  play  a  key  role  in  affecting  the 
performance  outcomes  of  each  program  as  well  as  its  subsequent  interdependent  partners. 
This  enabled  us  to  find,  for  example,  the  critical  nodes  and  interdependencies  in  the  system. 
As  a  consequence  of  this  work,  in  future  studies  we  can  create  a  hypothetical  breach  at  a 
node,  or  resource  cutoff  in  some  in-flow,  and  discover  its  likely  effects.  We  can  extend  that 
to  conjunctures  of  breaches  or  breaks  in  the  flows.  Similarly,  we  can  determine  the  most 
robust  and  weakest  programs  in  the  system  (i.e.,  those  most  and  least  likely  to  have 
breaches  or  fail).  We  can  also  use  the  model  to  examine  the  changes  to  the  system  that 
might  increase  its  robustness. 
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The  complexity  of  the  joint  environment  is  likely  to  have  consequences  related  to 
acquisition  activities.  The  precise  effect  on  acquisition,  and  its  resulting  managerial 
implications,  are,  as  of  yet,  unknown.  The  significance  of  the  research  is  three-fold: 

•  It  aims  to  forge  new  ground  in  identifying  the  effects  of  interdependency  on 
acquisition  and,  if  needed,  uncovering  early  indicators  of  interdependency  risk  so 
that  appropriate  governance  oversight  methods  can  then  be  isolated. 

•  It  provides  insight  into  the  nature  of  the  available  data  and  whether  it  can  support 
the  use  of  non-linear  methods  to  detect  and  prevent  cascading  consequences. 

•  It  leverages  a  decision-theoretic  model  that  captures  uncertainty  in  action 
outcomes  and  information  of  neighboring  nodes  to  describe  the  sequential 
decision  making  process  inherent  to  MDAPs. 

We  believe  that  given  the  frequency  with  which  government  agencies  are  moving 
toward  joint  initiatives,  the  findings  of  this  research  project  based  on  DoD  programs  may 
prove  instrumental  to  a  wide-range  audience. 

Research  Methodology 

To  perform  this  study,  we  designed  a  methodology  that  includes  four  goals.  We  first 
selected  a  small  subset  of  inter-related  MDAPs  based  on  a  set  of  criteria  to  form  our  case 
study.  We  defined  Goals  1  and  2  to  determine  whether  the  MDAP  data  in  the  form  of  the 
SARs,  DAES,  and  PEs  is  sufficient  to  identify  the  effects  of  interdependency  on  acquisition 
and  uncover  any  early  indicators  of  an  interdependency  risk.  These  goals  also  determine 
whether  a  decision-theoretic  model  in  Goal  3  is  a  feasible  next  step.  Having  verified  that  this 
is  the  case,  we  then  formulated  a  decision-theoretic  model.  Finally,  we  captured  the 
essence  of  the  data  acquisition  process  for  our  study  and  the  lessons  learned. 

Goal  1:  Identify  highly  dependent  parts  of  the  MDAP  network. 

•  What  are  the  essential  features  of  the  network  that  reveal  the  joint  space 
dynamics? 

•  What  are  the  relative  priorities  associated  with  these  features,  and  how  do  they 
affect  the  network  relationship? 

Goal  2:  Analyze  and  understand  the  data  available  from  MDAP  performance  reports 
to  extract  features  of  network  dynamics. 

•  What  are  the  local  issues  that  lead  toward  a  breach  or  near-breach  situation? 

•  How  often  and  why  do  the  local  mitigation  efforts  fail  to  improve  the 
performance? 

•  How  do  we  identify  the  non-local  issues  that  result  from  the  interdependencies? 

•  How  do  we  determine  the  cascading  effect  through  the  network? 

We  planned  to  approach  Goal  2  from  two  perspectives:  (1)  local  perspective,  where 
the  analyses  are  based  solely  on  the  individual  program’s  own  data;  and  (2)  non-local 
perspective,  where  the  analyses  are  based  on  the  data  of  MDAPs  existing  in  the  joint  space 
of  the  individual  program.  Lessons  learned  from  these  analyses  should  enable  the 
stakeholders  to  take  appropriate  measures  to  improve  the  performance  of  the  programs. 

Goal  3:  Formulate  a  decision-theoretic  model  that  harnesses  Decentralized  Markov 
Decision  Process  (DEC-MDP)  formalism. 

What  are  the  essential  characteristics  of  the  MDAP  network  that  justify  a  DEC- 
MDP  model? 

How  do  we  model  the  MDAP  network  as  a  decentralized  system? 
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•  What  are  the  key  challenges  in  the  design  of  the  DEC-MDP? 

•  What  essential  features  should  the  DEC-MDP  model  incorporate  for  better 
predictability? 

The  DEC-MDP  is  a  sub-class  of  decentralized  partially  observable  MDP  (DEC- 
POMDP;  Bernstein,  Givan,  Immerman,  &  Zilberstein,  2002),  which  we  propose  to  model  the 
behavior  of  the  MDAP  network.  A  state  is  a  snapshot  in  time  of  the  MDAP’s  status  that 
consists  of  crucial  local  and  non-local  information.  A  policy  is  a  mapping  from  a  state  to  an 
action.  This  formalism  would  allow  the  MDAP  to  execute  the  appropriate  local  policy  to 
achieve  higher  performance.  Our  aim  is  to  define  a  computationally  tractable  model. 

Goal  4:  Understand  the  characteristics  of  the  existing  data  resources. 

•  What  are  the  challenges  to  pre-process  the  existing  data? 

•  What  key  information  do  we  gain  from  the  existing  data? 

•  What  are  the  key  limitations  in  the  existing  data? 

•  What  are  the  data  requirements  to  design  a  complete  DEC-MDP  model? 

•  How  do  we  integrate  the  various  program-related  documents  in  a  coherent  and 
meaningful  fashion  to  aid  the  decision-makers  as  well  as  the  researchers  in 
building  complete  models? 

Goal  4  recommends  what  should  be  done  to  capture  information  so  that  the 
decision-making  process  becomes  efficient  and  complete. 

Findings 

Our  findings  indicate  that  MDAP-related  data  characteristics  support  the  multiple 
perspective  study  of  perturbations  and  it  is  possible  to  recast  the  study  of  cascading  effects 
as  a  sequential  decision  problem.  We  also  note  that  it  is  crucial  to  consider  the  uncertainty 
in  action  outcomes  in  the  decision-making  process  and  that  a  non-local  perspective  may 
help  explain  a  performance  breach  in  situations  where  a  solely  local  perspective  does  not. 
These  observations  provide  evidence  supporting  our  conjecture  that  MDPs  are  a  good 
avenue  to  study  interdependencies  in  the  MDAP  network  and  to  capture  early  indicators  of 
interdependency  risk.  Finally,  we  have  captured  the  informational  value  in  the  existing  data 
and  challenges  inherent  in  the  data  collection  process  with  respect  to  their  role  in  isolating 
risks  and  initiating  appropriate  government  oversight  methods. 

The  rest  of  the  paper  is  structured  as  follows:  in  the  Network  Model  section  we 
identify  the  network  dependencies  among  the  MDAPs  and  define  a  sample  network  for 
analyses  (Goal  1 );  in  the  Case  Study  of  MDAP_A  Funding  Network  section  we  investigate 
the  local  and  non-local  causes  for  degradation  in  performance  of  the  nodes  in  the  sample 
network  (Goal  2);  in  A  Decision-Theoretic  Model  for  MDAP  Network  section  we  present  the 
DEC-MDP  model  formulation  (Goal  3)  followed  by  observations  made  about  the 
characteristics  of  the  available  data  in  the  Understanding  the  Characteristics  of  the  Existing 
Data  section  (Goal  4);  finally,  the  Conclusions  and  Future  Work  section  concludes  with  the 
lessons  that  we  learn  through  this  process. 

Network  Model 

In  this  section,  we  first  enumerate  various  MDAP  performance  reports  (see  below) 
and  discuss  their  significance  in  light  of  networking  dependencies  among  the  MDAPs.  We 
also  define  a  sample  funding  network  from  our  chosen  MDAPs  in  an  effort  to  investigate  its 
performance.  Specifically,  we  define  a  process  to  choose  an  MDAP  program  to  be  the  focus 
of  our  investigation  and  identify  its  immediate  network  based  on  the  interrelationships  it 
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maintains  with  neighboring  MDAPs.  We  use  the  lessons  learned  from  the  analyses  of  this 
sample  network  to  build  an  accurate  decision-theoretic  model. 

Available  Data  Resources  on  MDAP  Performance 

The  information  pertaining  to  acquisition  research  is  overwhelming  and  multifarious. 

It  appears  to  be  a  daunting  task  for  the  acquisition  researchers,  let  alone  the  program 
managers,  to  integrate  and  understand  the  vast  and  dynamic  data  in  a  coherent  way.  To 
define  the  interrelationship  among  the  MDAPs  from  a  network-centric  viewpoint,  and  to 
identify  different  network  dependencies  within  the  domain  of  MDAPs,  the  following  set  of 
data  resources  are  useful: 

•  monthly  DAES  reports  that  provide  an  early-warning  report  on  the  status  of  some 
program  features  such  as  cost,  schedule,  performance,  funding,  and  so  forth; 

•  SARs  that  summarize  the  latest  estimates  of  cost,  schedule,  and  technical  status 
to  be  reported  annually  in  conjunction  with  the  president’s  budget;  and 

•  PE  documents  that  are  used  to  justify  congressional  budgeting  process. 

Types  of  Interdependent  Networks  Within  the  MDAPs  Domain 

In  addition  to  these,  the  program  managers  also  report  on  four  external 
interdependencies:  (1)  data  interdependencies  with  other  DoD  programs;  (2)  funding 
received  from  other  DoD  programs;  (3)  contractor  interdependencies;  and  (4) 
budgeting/spending  authority  interdependencies.  This  information  is  useful  in  identifying  four 
types  of  interdependency  networks  among  the  MDAPs.  As  an  example,  Figure  1  provides  a 
glimpse  of  the  data  interdependencies  of  the  78  MDAPs  in  2010.  In  terms  of  the  data 
interdependencies,  the  78  MDAPs  exhibit  a  total  of  989  data  interdependencies.  (Note  that 
MDAP  programs  will  have  interdependencies  with  non-MDAP  programs.  For  example,  the 
Joint  Strike  Fighter  program  identifies  data  and  funding  interdependencies  with  the  Italian, 
German,  and  French  defense  departments.)  In  the  current  dataset,  17%  of  the 
interdependencies  are  outbound,  37%  are  inbound,  and  45%  are  bidirectional.  Additionally, 
per  managerial  reports,  multiple  perspectives  may  be  critical  to  the  decision-making 
process.  In  most  cases,  the  boundaries  are  drawn  and  the  regions  thus  identified,  based  on 
the  entire  set  of  assets  that  are  transferred  or  exchanged  to  provide  a  given  capability.  As 
such,  the  flows  in  and  out  of  a  node  can  be  examined  to  conduct  scenario  planning  or  “what- 
if  analyses. 


Figure  1.  MDAP  Data  Interdependencies  in  2010 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-201  - 


Figure  2.  Funding  Network  of  MDAP_A 
MDAP_A  Funding  Network 

As  we  discussed  in  the  Introduction,  we  chose  to  do  a  case  study  because  of  the  the 
characteristics  of  the  data.  MDAP_A,  a  communications  program  initiated  in  2004,  whose 
program  name  has  been  scrubbed  for  confidentiality  purposes,  is  the  central  MDAP  for  our 
study.  This  program  is  our  focus  because  (a)  the  data  available  about  this  program  is 
significant;  and  (b)  between  the  years  2006  and  2010,  it  experienced  multiple  APB  breaches 
and  increase  in  %PAUC,  making  it  a  critical  node  for  reference.  Using  information  about  the 
funding  partners  of  MDAP_A,  we  defined  a  logical  funding  network  shown  in  Figure  2. The 
other  nodes  in  the  graph  are  neighbor  programs  of  MDAP_A  that  share  common  funding 
agencies.  The  funding  network  allows  us  to  do  a  detailed  study  of  the  performance  of  the 
member  nodes  and  to  understand  the  cascading  effects  described  in  the  Case  Study  of 
MDAP_A  Funding  Network  section.  In  the  future,  we  plan  to  apply  the  lessons  learned  from 
this  focused  study  to  the  entire  MDAP  network. 

Case  Study  of  MDAP_A  Funding  Network 

In  this  section,  we  analyze  the  data  that  we  gathered  from  the  available  performance 
reports  of  all  the  MDAPs  in  the  MDAP_A  funding  network,  from  the  local  and  non-local 
perspectives  as  defined  in  the  Introduction. 

Consider  the  funding  network  for  MDAP_A  in  Figure  2.  MDAP_A  lies  at  the  center  of 
this  undirected  network  that  contains  five  nodes.  The  link  between  any  two  nodes  refers  to 
the  funding  relationship  and  serves  as  interface  among  the  programs.  These  links  illustrate 
the  interdependent  regions  of  the  case  study  network.  We  analyze  the  performance  of  the 
programs  based  on  the  APB  breaches  and  amount  in  increase  in  %PAUC.  Five  types  of 
APB  breaches  are  reported  in  the  performance  reports,  which  are  schedule;  performance; 
research,  development,  test,  and  evaluation  (RDT&E);  procurement;  and  PAUC.  A  program 
is  considered  to  perform  poorly  if  it  experiences  frequent  APB  breaches  and/or  increase  in 
%PAUC. 

Suppose  that  the  central  program  MDAP_A  has  been  under-performing  for  a  period 
of  time,  and  also  assume  that  any  two  neighboring  programs  have  been  under-performing 
as  well  in  subsequent  periods.  We  want  to  understand  their  performance  degradation  by 
investigating  the  following  questions: 

Q1 :  What  are  the  local  reasons  for  a  program  (e.g.,  MDAP_A)  to  underperform? 

Q2:  How  often  and  why  do  the  forecasting  of  mitigation  efforts,  as  captured  in 
monthly  DAES  reports,  turn  out  to  be  ineffective? 

Q3.  What  are  the  non-local  reasons  for  poor  performance? 
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•  Q4:  How  does  the  effect  of  one  underperforming  program  propagate  through  the 
link  towards  a  neighbor  program  and  affect  it? 

•  Q5:  Why  is  a  program  that  is  performing  as  expected  not  affected  by  this 
perturbation? 

•  Q6:  How  does  this  network-centric  approach  facilitate  the  understanding  of  the 
underlying  problems  leading  to  cascade  in  breaches  and  help  the  stakeholders 
take  appropriate  measures? 

To  address  the  above  questions,  we  employed  the  following  three-phase  approach: 

•  Phase  1 :  Identify  programs  in  the  MDAP_A  funding  network  that  underperform  by 
analyzing  SAR  files  of  all  programs  specifically  for  information  pertaining  to  APB 
breaches  and  increase/decrease  in  %PAUC. 

•  Phase  2:  Study  the  local  reasons  for  the  poor  performance  of  the  programs 
based  on  their  respective  DAES  reports. 

•  Phase  3:  Study  the  non-local  reasons  for  poor  performance  by  analyzing  the 
SAR  files. 


In  the  remainder  of  this  section,  we  discuss  the  details  of  this  three-phase  approach. 

Phase  1:  Identify  Programs  in  the  MDAP_A  Funding  Network  That  Exhibit  Poor 
Performance 

We  studied  the  yearly  performance  of  MDAP_A  funding  network  using  the  SAR  files. 
Table  1  shows  the  APB  breaches  and  %PAUC  during  2004-2010  for  the  nodes  in  the 
MDAP_A  network.  Programs  initiated  after  2004  have  data  from  their  respective  start  date. 

Table  1.  SAR  Summary  of  the  MDAP_A  Funding  Network  for  2004-2010 


MDAP  A 


Schedule 


APB 

Breach 

Performance  RDT&E  Procurement  PAUC 


2004 

2005 

2006 
2007 
2009 


2010 

MDAP  B 


2004 

2005 

2006 
2007 


None 


None 


None 


None 


None 


None 


None 


None 


None 


None 


None  (-9.98%) 
None  (-1 1 .65%) 
None  (-6.14%) 
None  (-1 .24%) 
None  (3.14%) 
None  (3.82%) 


None 

None  (3.85%) 
None  (3.85%) 
None  (7.69%) 
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MDAP  D 


None  (-26.92%) 
None  (-19.23%) 


None  (6.51% 


None  (0.93%) 


None  (-37.79%) 
None  (-26.75%) 


2009 

None 

None 

None 

None 

None  (2.45%) 

2010 

None 

None 

None 

None  (1.05%) 

MDAP E 

2006 

None 

None 

None 

None 

None  (- 
10.685%) 

2007 

None 

None 

None 

None 

None  (-4.81%) 

2009 

None 

None 

None 

None 

None  (-3.98%) 

2010 

None 

None 

None 

None 

None  (-1 1 .24%) 

In  SAR  files,  APB  breach  is  defined  as  a  condition  in  which  the  value  of  the 
respective  breach  parameters  (schedule,  performance,  RDT&E,  procurement,  and  PAUC)  is 
in  the  range  of  1 0%— 1 5%,  beyond  which  the  condition  is  defined  as  a  Nunn-McCardy 
breach.  Table  1  captures  whether  a  program  has  APB  breaches  in  a  given  year  and  what  is 
the  %PAUC  of  that  program.  A  program  may  have  more  than  one  APB  breach  but 
experience  a  decrease  in  %PAUC.  For  example,  in  the  year  2006,  the  program  MDAP_A 
experienced  schedule,  RDT&E,  and  performance  breaches,  yet  its  %PAUC  decreased.  Two 
possible  reasons  could  account  for  this  fact:  (1)  the  decrease  in  %PAUC  could  be  due  to 
lagging  effect  from  previous  year;  and/or  (2)  according  to  project  management  triangle 
model  (Bethke,  2003),  program  managers  may  intentionally  choose  biases  towards  better 
performance  of  one  component  of  the  program  by  trading  it  off  with  performance  of  other 
components. 

Table  1  indicates  that  MDAP_A,  MDAP_B  and  MDAP_C  programs  have  been 
experiencing  frequent  APB  breaches  and  increase  in  %PAUC  during  2004  and  2010. We 
intend  to  understand  the  causes  of  poor  performance  for  these  programs  in  the  Phase  2: 
Investigation  of  Local  Reasons  for  Poor  Performance  subsection.  Of  the  three  poorly 
performing  programs,  we  chose  to  analyze  MDAP_A  and  MDAP_B.  We  identified  the  local 
causes  for  these  two  programs  and  then  determined  whether  interdependency  issues 
existed  among  them.  In  other  words,  in  the  subsection  Phase  3:  Study  the  Non-Local 
Reasons  for  Poor  Performance  by  Analyzing  the  SAR  we  observed  whether  any  of  these 
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“poorly  performed”  programs  propagate  their  performance  effects  to  the  other  program, 
causing  the  other  programs  to  perform  poorly  as  well. 

Phase  2:  Investigation  of  Local  Reasons  for  Poor  Performance 

In  this  subsection,  we  investigate  the  performance  issues  local  to  individual  MDAPs 
and  also  track  how  effective  “mitigation  forecasting”  is  to  resolve  pertaining  issues. 

We  used  the  DAES  reports  of  individual  programs  to  analyze  their  performance  from 
a  local  perspective.  We  observed  that  the  DAES  reports  capture  the  performance  issues  of 
a  program’s  local  domain.  We  focused  on  four  performance  issues  recorded  in  the  DAES 
reports,  namely,  cost,  schedule,  performance,  and  funding. 

Understanding  the  Local  Causes  for  MDAP_A  to  Perform  Poorly 

We  studied  a  total  of  40  available  DAES  reports  for  MDAP_A  between  2006  and 
2010.  These  reports  are  published  monthly  each  year,  including  the  election  year  of  2008, 
unlike  the  SAR,  which  did  not  report  in  2008.  The  program  status  is  presented  in  DAES 
reports  through  the  following  parameters:  cost,  schedule,  funding,  performance,  and  life 
cycle  sustainment.  We  focus  on  cost,  schedule,  performance,  and  funding  parameters.  Each 
parameter  reflects  both  the  APB  and  contract  status.  The  status  for  each  month  is 
represented  in  one  of  three  colors  depending  on  the  severity  of  the  pertaining  issue.  Green 
reflects  the  normal  state  meeting  all  requirements,  while  yellow  reflects  resolvable  issues 
(resolvable  APB/contract),  and  red  refers  to  a  state  that  could  not  meet  the  requirements 
(critical  APB/contract). 

We  first  understood  how  effective  the  APB  and  contract  forecasting  were  to  mitigate 
the  pertinent  problems  by  (1 )  recording  the  instances  where  the  forecasting  was  effective, 
as  well  as  where  it  was  not  ineffective;  and  (2)  identifying  the  issues  that  caused  the 
predictions  to  slip.  We  then  analyzed  the  issues  for  deeper  understanding  and 
categorization. 

We  present  our  analyses  in  tabular  format  for  three  parameters:  cost,  schedule,  and 
funding  in  the  three  following  subsections.  Because  MDAP_A  did  not  have  any  performance 
issues,  we  focused  on  the  cost,  schedule,  and  funding  issues. 

MDAP_A  Cost  Analysis 


Table  2  captures  cost-related  issues  for  the  program. 

Table  2.  MDAP_A  Cost  Analysis  Using  DAES  Reports  From  2006  to  2010 


Current  Status 

Status  at  the  Predicted 
Month 

Causes 

Month:  April  2007 

Issue:  Contract  -  Yellow 

Mitigation  forecast:  8  months 

Month:  September  2007 
Status:  Contract  -  Red 
Note:  After  5  months  the 
contract  issue  turns  into 
critical 

Issue  1 :  Hardware 
building 

Issue  2:  Hardware 
design 

Issue  3:  Logistics  issue 

Month:  September  2007 

Issue:  Contract  -  Red 

APB  -  Yellow 

Mitigation  forecast: 

8  months 

Month:  May  2008 

Status:  Contract  -  Yellow 
APB-Green 

Issue  1-3:  Resolved 
Issue  4:  Contractor 
unable  to  forecast  cost 

Month:  May  2008 

Month:  July  2008 

Issue  4:  Contractor 
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Issue:  Contract  -  Yellow 

Mitigation  forecast:  8  months 

Status:  Contract  -  Red 
Note:  After  5  months  the 
contract  issue  turns  into 
critical 

unable  to  forecast  cost 
Issue  5:  Schedule  delav 
increased  contract  cost 

Month:  July  2008 

Issue:  Contract  -  Red 

Mitigation  forecast:  4  months 

Month:  November  2008 
Status:  Contract  -  Red 

Issue  4:  Contractor 
unable  to  forecast  cost 
Issue  5:  Schedule  delav 
increased  contract  cost 

Month:  November  2008 

Issue:  Contract  -  Red 

Mitigation  forecast:  1  month 

Month:  December  2008 
Status:  Contract  -  Red 

Issue  4:  Contractor 
unable  to  forecast  cost 
Issue  5:  Schedule  delav 
increased  contract  cost 

Month:  December  2008 

Issue:  Contract  -  Red 

Mitigation  forecast: 

2  months 

Month:  February  2009 
Status:  Contract  -  Red 

Issue  4:  Contractor 
unable  to  forecast  cost 
Issue  5:  Schedule  delav 
increased  contract  cost 

Month:  February  2009 

Issue:  Contract  -  Red 

Mitigation  forecast:  4  months 

Month:  June  2009 

Status:  Contract  -  Yellow 

Issue  4:  Remains 

Issue  5:  Remains 

Month:  June  2009 

Issue:  Contract  -  Yellow 

Mitigation  forecast:  8  months 

Month:  February  2010 
Status:  Contract  -  Yellow 

Issue  4:  Remains 

Issue  5:  Remains 

Month:  February  2010 

Issue:  Contract  -  Yellow 

Mitigation  forecast:  1  month 

Month:  March  2010 

Status:  Contract  -  Yellow 

Issue  4:  Remains 

Issue  5:  remains 

Month:  March  2010 

Issue:  Contract  -  Yellow 

Mitigation  forecast:  2  months 

Month:  April  2010 

Status:  Contract  -  Yellow 
Note:  May  2010  report  is 
incomplete 

Issue  4:  Remains 

Issue  5:  Remains 

Lessons  Learned.  Although  Table  2  suggests  that  there  are  some  instances  where 
the  forecasting  turned  out  to  be  effective,  we  observed  and  focused  on  the  instances  where 
the  cost-related  forecasting  was  not  effective.  We  identified  two  local  issues,  namely,  (1) 
contractors  inability  to  forecast  cost,  and  (2)  schedule  delay  leading  to  increased  contract 
cost,  which  appear  to  recur  and  lead  to  increased  program  costs. 

MDAP_A  Schedule  Analysis 


Table  3  captures  schedule-related  issues  for  the  program. 

Table  3.  MDAP_A  Schedule  Analysis  Using  DAES  Reports  From  2006  to  2010 


Current  Status 

Status  at  the  Predicted 

Month 

Causes 

Month:  June  2007 

Issue:  Contract  -  Yellow 
Mitigation  forecast:  2 
months 

Month:  September  2007 

Status:  Contract  -  Yellow 

Note:  August  2007  report  is 
not  available 

Issue  1:  Delav  in  MOU 
sign  with  Australia 
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Month:  September  2007 
Issue:  Contract  -  Yellow 
Mitigation  forecast:  1  month 

Month:  October  2007 

Status:  Contract  -  Green 

Issue  1:  Remains 

Issue  2:  Software 
testing,  delivery,  and 
other  waveform  issues 

Month:  October  2007 

Issue:  APB  -  Yellow 

Mitigation  forecast:  1  month 

Month:  November  2008 

Status:  APB  -  Green 

Issue  1:  Resolved 

Issue  2:  Resolved 

Month:  March  2008 

Issue:  APB  -  Yellow 

Contract  -  Yellow 
Mitigation  forecast:  8 
months 

Month:  November  2008 

Status:  APB  -  Red 

Contract  -  Red 

Issue  3:  Hardware 
testing  and 
performance  failure 
Issue  4:  Execution 
delay  in  contractor’s 
schedule  &  lack  in 
funding 

Month:  November  2008 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  forecast:  2 
months 

Month:  December  2008 

Status:  APB  -  Red 

Contract  -  Red 

Issue  3:  Hardware 
testing  and 
performance  failure 
Issue  4:  Execution 
delay  in  contractor’s 
schedule  &  lack  in 
funding 

Month:  December  2008 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  forecast:  2 
months 

Month:  February  2009 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  Forecast: 

2  months 

Issue  3:  Hardware 
testing  and 
performance  failure 
Issue  4:  Execution 
delay  in  contractor’s 
schedule  &  lack  in 
funding 

Month:  February  2009 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  forecast:  4 
months 

Month:  June  2009 

Status:  APB  -  Red 

Contract  -  Red 

Issue  4:  Execution 
delay  in  contractor’s 
schedule  &  lack  in 
funding 

Issue  3:  Hardware 
testing  and 
performance  failure 

Month:  June  2009 

Issue:  APB  -  Red 

Contract  -  Yellow 
Mitigation  forecast: 

APB:  4  months 

Contract:  8  months 

Month:  October  2009 

Status:  APB  -  Green 

Contract  -  Yellow 

Month:  October  2009 

Issue:  Contract  -  Yellow 
Mitigation  forecast: 

Contract:  5  months 

Month:  March  2010 

Status:  APB  -  Red 

Contract  -  Yellow 

Issue  4:  Execution 
delay  in  contractor’s 
schedule  &  lack  in 
funding 

Issue  3:  Hardware 
testing  and 
performance  failure 
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Lessons  Learned.  Although  there  are  some  instances  for  which  the  forecasting 
turned  out  to  be  effective,  we  observed  and  focused  on  the  instances  where  the  schedule- 
related  forecasting  was  not  effective.  We  identify  two  local  issues,  namely,  (1)  hardware 
testing  and  performance  failure,  and  (2)  execution  delay  and  lack  of  funding  that  appear  to 
recur  and  lead  the  program  towards  schedule  delay. 

MDAP_A  Funding  Analysis 

Table  4  captures  funding  related  issues  for  the  program. 


Table  4.  MDAP_A  Funding  Analysis  Using  DAES  Reports  From  2006  to  2010 


Current  Status 

Status  at  the  Predicted 

Month 

Causes 

Month:  April  2007 

Issue:  APB  -  Yellow 

Mitigation  forecast: 

Contract:  5  months 

Month:  September  2007 

Status:  APB  -  Yellow 

Issue  1:  WPN  fund  cut 

Month:  September  2007 
Issue:  APB  -  Yellow 

Mitigation  forecast: 

Contract:  1  month 

Month:  October  2007 

Status:  APB  -  Yellow 

Issue  1:  WPN  fund  cut 

Month:  October  2008 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  forecast: 

APB:  4  months 

Contract:  2  months 

Month:  December  2008 

Status:  APB  -  Red 

Contract  -  Red 

Issue  1:  WPN  fund  cut 

Month:  December  2008 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  forecast: 

APB:  4  months 

Contract:  2  months 

Month:  February  2009 

Status:  APB  -  Red 

Contract  -  Red 

Issue  1:  WPN  fund  cut 

Month:  February  2009 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  forecast: 

APB:  1  month 

Contract:  1  month 

Month:  March  2009 

Status:  APB  -  Green 

Contract  -  Green 

Month:  April  2009 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  forecast: 

By  the  current  month 

Month:  May  2009 

Status:  APB  -  Red 

Contract  -  Red 

Issue  1:  WPN  fund  cut 

Month:  May  2009 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  forecast: 

4  months 

Month:  September  2009 

Status:  APB  -  Green 

Contract  -  Green 

Lessons  Learned.  Although  there  are  some  instances  for  which  the  forecasting 
turned  out  to  be  effective,  we  observed  and  focused  on  the  instances  where  the  funding- 
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related  forecasting  was  not  effective.  We  identified  one  local  issue,  namely  the  weapons 
procurement  cut  (WPN),  that  appears  to  recur  and  lead  the  program  towards  experiencing 
funding-related  problems  (for  example,  lack  of  funding  caused  schedule  delay,  as  captured 
in  the  MDAP_A  schedule  analyses  section). 

Based  on  the  above  lessons  from  the  cost,  schedule,  and  funding  analyses  of 
MDAP_A,  we  identified  the  following  observations  that  appear  to  be  responsible  for  APB 
cost  and  schedule  breach  of  MDAP_A: 

•  Observation  1 :  Design  of  MDAP_A  relies  on  cutting  edge  technology.  It  seems 
that  the  contractor  underestimated  or  could  not  accurately  estimate  the  technical 
challenges  and  the  amount  of  funding  required  to  accomplish  the  tasks. 

•  Observation  2:  MDAP_A  suffered  greatly  due  to  budget  cuts.  The  program  did 
not  receive  required  amount  of  funding  from  the  government  (congressional 
committee),  which  delayed  the  schedule,  and  as  a  consequence  cost  increased. 

Understanding  the  Local  Causes  for  MDAP_B  to  Perform  Poorly 

We  studied  a  total  of  44  available  DAES  reports  for  MDAP_B  between  2006  and 
2010.  We  first  understood  the  effectiveness  of  APB  and  contract  forecasting  to  mitigate  the 
pertinent  problems.  We  did  this  by  recording  the  instances  when  the  forecasting  was 
effective,  as  well  as  when  it  was  not  ineffective.  We  then  sought  to  identify  and  analyze  the 
issues  that  caused  the  predictions  to  slip. 

We  present  our  analyses  in  tabular  format  for  three  parameters:  cost,  schedule,  and 
funding  in  three  following  subsections.  Because  MDAP_B  did  not  have  any  performance 
issues,  we  focused  on  cost,  schedule,  and  funding  issues. 

MDAP_B  Cost  Analysis 


Table  5  captures  cost-related  issues  for  the  program. 

Table  5.  MDAP_B  Cost  Analysis  Using  DAES  Reports  From  2006  to  2010 


Current  Status 

Status  at  the  Predicted  Month 

Causes 

Month:  February  2007 
Issue:  APB  -  Yellow 
Mitigation  forecast:  3 
months 

Month:  May  2007 

Status:  APB  -  Yellow 

Issue  1:  Require 

procurement 

funding 

Month:  May  2007 

Issue:  APB  -  Yellow 
Mitigation  forecast:  1 
month 

Month:  June  2007 

Status:  APB  -  Yellow 

Issue  1:  Require 

procurement 

funding 

Month:  June  2007 

Issue:  APB  -  Yellow 
Mitigation  forecast:  2 
months 

Month:  August  2007 

Status:  APB  -  Yellow 

Issue  1:  Require 

procurement 

funding 

Issue  2:  Contractor 
cost  increase 

Month:  August  2007 

Issue:  APB  -  Yellow 
Contract  -  Yellow 
Mitigation  forecast:  1 
month 

Month:  September  2007 

Status:  APB  -  Yellow 

Contract  -  Yellow 

Issue  1:  Require 

procurement 

funding 

Issue  2:  Contractor 
cost  increased 
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Month:  September  2007 
Issue:  APB  -  Yellow 
Contract  -  Yellow 
Mitigation  forecast: 

APB:  1  month 

Contractor:  8  months 

Month:  October  2007 

Status:  APB  -  Green 

Contract  -  Yellow 

Issue  1:  Require 

procurement 

funding 

Issue  2:  Contractor 
cost  increased 

Month:  October  2007 
Issue:  Contract  -  Yellow 
Mitigation  forecast:  8 
months 

Month:  June  2008 

Status:  Contract  -  Yellow 

Issue  1:  Require 

procurement 

funding 

Issue  2:  Contractor 
cost  increased 

Month:  June  2008 

Issue:  Contract  -  Yellow 
Mitigation  forecast:  8 
months 

Month:  February  2009 

Status:  Contract  -  Yellow 

Issue  1:  Require 

procurement 

funding 

Issue  2:  Contractor 
cost  increased 

Month:  February  2009 
Issue:  Contract  -  Yellow 
Mitigation  forecast:  8 
months 

Month:  October  2009 

Status:  Contract  -  Yellow 

Issue  1:  Require 

procurement 

funding 

Issue  2:  Contractor 
cost  increased 

Month:  October  2009 
Issue:  Contract  -  Yellow 
Mitigation  forecast:  8 
months 

Month:  April  2010 

Status:  Contract  -  Yellow 

Note:  No  data  available  beyond 
April  2010 

Issue  1:  Require 

procurement 

funding 

Issue  2:  Contractor 
cost  increased 

Lessons  Learned.  Although  there  are  some  instances  for  which  the  forecasting 
turned  out  to  be  effective,  we  observed  and  focused  on  the  instances  where  the  cost-related 
forecasting  was  not  effective.  We  identified  two  local  issues,  namely,  (1)  the  lack  in 
procurement  funding,  and  (2)  increased  contract  costs  that  appear  to  recur  and  lead  the 
program  towards  cost  increase. 

MDAP_B  Schedule  Analysis 

Table  6  captures  schedule  issue  for  the  program. 


Table  6.  MDAP_B  Schedule  Analysis  Using  DAES  Report  From  2006  to  2010 


Forecasting 

Status  at  the  Predicted 

Month 

Causes 

Month:  September  2006 

Issue:  APB  -  Yellow 

Mitigation  forecast:  6  months 

Month:  March  2007 

Status:  APB  -  Yellow 

Month:  March  2007 

Issue:  APB  Yellow 

Mitigation  forecast:  2  months 

Month:  May  2007 

Status:  APB  -  Yellow 

Month:  May  2007 

Issue:  APB  Yellow 

Mitigation  forecast:  1  month 

Month:  June  2007 

Status:  APB 

Month:  June  2007 

Issue:  APB  -  Yellow 

Mitigation  forecast:  2  months 

Month:  August  2007 

Status:  APB  -  Yellow 
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Month:  August  2007 

Issue:  APB  -  Yellow 

Mitigation  forecast:  1  month 

Month:  September  2007 

Status:  APB  -  Yellow 

Month:  September  2007 

Issue:  APB  -  Yellow 

Mitigation  forecast:  2  months 

Month:  November  2007 

Status:  APB  -  Green 

Month:  November  2010 

Issue:  APB  -  Yellow 

Mitigation  forecast:  5  months 

Month:  April  2010 

Status:  APB  -  Red 

Note:  No  data  available 
beyond  April  2010 

Issue  2:  Phase  1 
(Rifleman  Radio) 
Milestone  C 
decision  date 
postponement  and 
potential  to  move 
right  beyond 
threshold  date; 

A  MS  C  Threshold 
Breach  causes  the 
Phase  1  (AN/PRC- 
1 54)  to  be  Red 
through  +3  months 

Lessons  Learned.  Although  there  are  some  instances  for  which  the  forecasting 
turned  out  to  be  effective,  we  observed  and  focused  on  the  instances  where  the  schedule- 
related  forecasting  was  not  effective.  We,  however,  could  not  identify  the  issues  that  caused 
schedule  delay  for  MDAP_B. 

MDAP_B  Funding  Analysis 

Table  7  captures  funding  related  issues  for  the  program. 


Table  7.  MDAP_B  Funding  Analysis  Using  DAES  Report  From  2006  to  2010 


Forecasting 

Status  at  the  Predicted 
Month 

Causes 

Month:  February  2007 

Issue:  APB  -  Yellow 

Contract  -  Yellow 

Mitigation  forecast:  3  months 

Month:  May  2007 

Status:  APB  -  Yellow 

Contract  -  Yellow 

Issue  1:  Require 
procurement  funding 

Month:  May  2007 

Issue:  APB  -  Yellow 

Contract  -  Yellow 

Mitigation  forecast:  1  month 

Month:  June  2007 

Status:  APB  -  Yellow 

Contract  -  Yellow 

Issue  1:  Require 
procurement  funding 

Month:  June  2007 

Issue:  APB  -  Yellow 

Contract  -  Yellow 

Mitigation  forecast:  2  months 

Month:  August  2007 

Status:  APB  -  Yellow 

Contract  -  Yellow 

Issue  1:  Require 
procurement  funding 

Month:  August  2007 

Issue:  APB  -  Yellow 

Contract  -  Yellow 

Mitigation  forecast:  1  month 

Month:  September  2007 
Status:  APB  -  Green 

Contract  -  Green 

Month:  March  2008 

Issue:  APB  -  Yellow 

Mitigation  forecast:  4  months 

Month:  July  2008 

Status:  APB  -  Yellow 

Issue  1:  Require 
procurement  funding 
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Month:  July  2008 

Issue:  APB  -  Yellow 

Mitigation  forecast:  3  months 

Month:  October  2008 

Status:  APB  -  Yellow 

Issue  1:  Require 
procurement  funding 

Month:  October  2008 

Issue:  APB  -  Yellow 

Mitigation  forecast:  3  months 

Month:  January  2009 

Status:  APB  -  Yellow 

Issue  1:  Require 
procurement  funding 

Month:  January  2009 

Issue:  APB  -  Yellow 

Mitigation  forecast:  current 
month 

Month:  February  2009 

Status:  APB  -  Yellow 

Issue  1:  Require 
procurement  funding 

Month:  February  2009 

Issue:  APB  -  Yellow 

Mitigation  forecast:  current 
month 

Month:  March  2009 

Status:  APB  -  Green 

Month:  June  2009 

Issue:  APB  -  Yellow 

Mitigation  forecast:  5  months 

Month:  November  2009 
Status:  APB  -  Red 

Issue  2:  R&D 
shortfall  driven  by 
overall  technical  and 
schedule  issues 

Issue  3:  Hardware 
testing  issue  to 
increase  program 
cost 

Month:  November  2009 

Issue:  APB  -  Red 

Mitigation  forecast:  8  months 

Month:  April  2010 

Status:  APB  -  Red 

Note:  No  data  available 
beyond  April  2010 

Issue  2:  FY  12-15 

R&D  shortfall  driven 
by  overall  technical 
and  schedule  issues 
Issue  3:  Hardware 
testing  issue  to 
increase  program 
cost 

Lessons  Learned.  Although  there  are  some  instances  for  which  the  forecasting 
turned  out  to  be  effective,  we  observed  and  focused  on  the  instances  where  the  funding- 
related  forecasting  was  not  effective.  We  identified  three  local  issues,  namely,  (1)  the 
requirement  of  procurement  funding,  (2)  R&D  shortfall  driven  by  overall  technical  and 
schedule  issues,  and  (3)  hardware  testing  issue  to  increase  program  cost.  These  issues 
appear  to  recur  and  lead  the  program  towards  experiencing  funding  related  problem  (for 
example,  cost  increase  as  captured  in  the  MDAP_B  cost  analyses  section). 

Based  on  the  above  lessons  learned  from  the  cost,  schedule  and  funding  analyses  of 
MDAP_B,  we  made  the  following  observations  about  what  is  responsible  for  APB  cost  and 
schedule  breach  of  MDAP_B: 

•  Observation  3:  Lack  in  procurement  funding  is  the  most  beleaguering  issue  for 
MDAP_B  for  its  observed  cost  and  funding  problems. 

•  Observation  4:  The  above  DAES  report-based  analyses,  however,  do  not  provide 
any  clue  for  shortfall  in  funding.  This  underscores  the  importance  of  looking 
beyond  the  local  view  of  a  program  and  to  search  for  non-local  causes  that  could 
have  contributed  to  the  degradation  in  performance.  This  motivated  us  to 
investigate  the  interdependent  region  between  MDAP_A  and  MDAP_B  to  identify 
possible  cascading  effects. 
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Phase  3:  Study  the  Non-Local  Reasons  for  Poor  Performance  by  Analyzing  the  SAR 

In  Table  8,  we  provide  a  summary  of  our  findings  from  our  study  of  DAES  reports  for 
MDAP_A  and  MDAP_B,  in  an  effort  to  understand  the  non-local  issues. 


Table  8.  MDAP_A  and  MDAP_B  Local  Issue  Summary  for  2006  to  2010 


MDAP A  Issues 

MDAP B  Issues 

•  Contractors  inability  to  forecast  cost 

•  Lack  in  procurement  funding 

•  Schedule  delay  increased  contract 

•  Increased  contract  cost 

cost 

•  Require  procurement  funding 

•  Hardware  testing  and  performance 

•  R&D  shortfall  driven  by  overall 

failure 

technical  and  schedule  issues 

•  Execution  delay  and  lack  of  funding 

•  Hardware  testing  issue  to  increase 
program  cost 

Table  8  indicates  that  although  contractor’s  ineffective  forecasting  and  schedule 
delays  (due  to  hardware  and  design  issues)  led  MDAP_A  to  incur  cost  overrun,  lack  in 
procurement  funding  appears  to  be  the  plaguing  issue  for  increase  in  cost  of  MDAP_B. 
Based  on  this  observation  we  propound  the  following  hypothesis:  The  cost  increase  of 
MDAP_A  in  year  2009  could  have  caused  procurement  funding  shortfall  for  MDAP_B  in 
2010,  which  in  effect  increased  the  cost  of  MDAP_B  (as  the  DAES  reports  on  MDAP_B 
suggest). 

To  verify  the  hypothesis,  we  prepared  the  following  two  tables  of  the  funding 
summary  (based  on  base  year  dollar)  from  the  SAR  files  of  MDAP_A  and  MDAP_B  for  the 
period  2004-2010.  Our  study  indicated  that  a  comparative  analysis  of  SAR  files  for  the 
programs  of  MDAP_A  funding  network  provides  insight  about  the  joint  space  and  hence  is 
useful  for  us  in  indentifying  the  non-local  issues.  SAR  captures  the  yearly  APB  breach 
status,  %PAUC,  cost  and  funding  data;  hence,  it  is  suitable  for  quantitative  analyses. 


Table  9.  MDAP_A  SAR  Funding  Summary  ($BY)  for  the  Period  2004-2010 


MDAP_A 

Baseline 

Quantity 

Current 

Quantity 

%PAUC 

Current  Year 
Required  Funding 
(x) 

Received 

Funding 

(y) 

Delta 

(y-x) 

2004 

6 

6 

-9.98 

221.1 

2005 

6 

6 

-11.65 

598.5 

579.8 

-18.7 

2006 

6 

6 

-6.14 

1012.1 

997.3 

-14.8 

2007 

6 

6 

-1.24 

1588.4 

1574.6 

-13.8 

2009 

6 

6 

3.14 

3163.2 

3006.3 

-156.9 

2010 

6 

6 

3.82 

3750.7 

3813.2 

62.5 

Table  10.  MDAP_B  SAR  Funding  Summary  ($BY)  for  the  Period  2004-2010 


MDAP_B 

Baseline 

Quantity 

Current 

Quantity 

%PAUC 

Current  Year 
Required  Funding 
(x) 

Received 

Funding 

(y) 

Delta 

(y-x) 

2004 

329574 

329574 

0 

44.2 

44.2 

0 

2005 

329574 

328514 

3.85 

137.2 

135.5 

-1.7 

2006 

329574 

328514 

3.85 

255.5 

250.3 

-5.2 
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2007 

329574 

95961 

7.69 

350.5 

348.1 

-2.4 

2009 

329574 

215961 

-26.92 

644.1 

593.2 

-50.9 

2010 

329574 

221978 

-19.23 

751.6 

711.1 

-40.5 

In  Tables  9  and  10,  we  focused  on  the  parameter  “delta,”  which  captures  the 
difference  in  the  amount  of  required  and  received  funding  for  the  respective  year.  For 
MDAP_A,  we  noticed  that  from  2009  to  2010,  the  %PAUC  increased  while  delta  turned  out 
to  be  positive.  On  the  other  hand,  for  MDAP_B,  from  2009  to  2010,  delta  retained  a  large 
negative  value,  even  though  given  the  trends  over  the  years,  the  increase  in  quantity  (~4000 
units)  is  not  large  enough  to  justify  this  increase.  Both  the  DAES  and  SAR  files  of  MDAP_B 
did  not  provide  reasons  for  the  large  negative  value  of  delta  in  2009  and  2010.  We  suspect 
that  the  cost  overrun  of  MDAP_A  in  2009  onwards  might  have  affected  MDAP_B  in  2010 
through  a  procurement  funding  short  fall.  This  observation,  even  if  it  may  not  be  conclusive, 
suggests  cascading  effects  between  neighboring  MDAPs.  We  believe  that  a  thorough  study 
of  the  entire  set  of  MDAPs  may  enable  us  to  find  more  interesting  interdependencies  and 
would  be  able  to  predict  the  flow  of  the  cascading  effects. 

Observations  from  the  Performance  Reports-Based  Analyses 

In  Subsection  Phase  2:  Investigation  of  Local  Reasons  for  Poor  Performance  and 
subsection  Phase  3:  Study  the  Non-Local  Reasons  for  Poor  Performance  by  Analyzing  the 
SAR,  we  studied  the  available  DAES  and  SAR  files  of  MDAP_A  and  MDAP_B  from  2006  to 
2010  in  an  effort  to  identify  cascading  effect  in  the  MDAP_A  funding  network.  We  tried  to 
understand  the  local  as  well  as  non-local  issues  that  led  the  programs  towards  breach 
condition.  The  following  is  the  summary  of  observations  from  this  process: 

•  Observation  1 :  Design  of  MDAPs  relies  on  cutting  edge  technology.  It  appears 
that  the  contractor  either  underestimates  or  cannot  accurately  estimate  the 
technical  challenges  and  the  amount  of  funding  required  to  accomplish  the  tasks. 

•  Observation  2:  Programs  are  observed  to  suffer  greatly  because  of  budget  cuts. 
Sometimes  programs  do  not  receive  required  amount  of  funding  from  the 
government  (congressional  committee),  which  delays  the  schedule,  and  as  a 
consequence  cost  increases. 

•  Observation  3:  Lack  in  procurement  funding  is  another  cause  that  leads  to  cost 
and  funding  problems. 

•  Observation  4:  Analyses  of  the  local  issues  and  the  fact  that  some  of  the  issues 
are  recurrent  indicate  that  either  the  root  cause  of  the  problem  is  not  captured  in 
the  DAES  documents  or  that  the  cause  is  exogenous  of  the  program  boundary. 

•  Observation  5:  Analyses  of  SAR  files,  on  the  other  hand,  offer  some  insights 
about  the  interdependency  of  the  programs. 

•  Observation  6:  The  observed  instance  of  possible  cascading  effect  in  the 
MDAP_A  network  (in  the  Phase  3:  Study  the  Non-Local  Reasons  for  Poor 
Performance  by  Analyzing  the  SAR  section)  motivates  us  to  design  an 
automated  scheme  that  would  be  able  to  identify  and  predict  the  likelihood  of 
cascading  effects. 

A  Decision-Theoretic  Model  for  MDAP  Networks 

A  Markov  Decision  Process  (MDP;  Bertsekas,  1987)  is  a  probabilistic  model  for 
decision-making  and  planning.  It  uses  dynamic  programming  to  decide  on  the  optimal 
actions  (in  this  case,  “cut  funding  by  50%”  or  “delay  schedule  by  six  months”)  that  yield  the 
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highest  expected  utility  (for  example,  no  PAUC  growth  or  no  APB  breaches).  MDPs  capture 
the  essence  of  sequential  processes  and  are  used  to  compute  decision  policies  that  lead  to 
best  long-term  performance  for  the  entire  network. 

In  theory,  MDPs  implement  two  forms  of  hedging  that  can  allow  managers  (1)  to  test 
their  decisions  to  avoid  the  possibility  of  failure,  and  (2)  to  choose  actions  that  ensure  higher 
overall  expected  reward.  These  hedging  strategies  alter  expectations  about  future  problems 
in  a  manner  that  allows  managers  to  shift  behaviors  to  improve  performance. 

In  our  approach,  MDAPs  are  considered  as  individual  agents  that  are  part  of  a 
cooperative  multi-agent  system,  and  decision-making  in  an  MDAP  network  is  viewed  as  a 
multi-agent  sequential  decision  problem  because  the  utility  gained  by  each  agent  depends 
on  a  sequence  of  actions  over  time.  Our  goal  is  to  determine  the  behavior  of  the  agents  that 
best  balances  the  risks  and  rewards  while  acting  in  an  uncertain  environment  with  stochastic 
actions. 

Each  MDAP  makes  its  individual  decisions  in  an  environment  where  the  state  space 
is  not  fully  observable,  meaning  that  the  nodes  in  the  network  (the  programs)  do  not  exactly 
know  which  state  they  are  in  at  any  particular  instant  because  they  do  not  have  complete 
information  about  their  neighbors.  With  the  partial-state  information,  the  individual  agents 
aim  to  optimize  the  joint  reward  function.  This  class  of  problems  is  modeled  as  a 
decentralized  partially  observable  MDAP  (DEC-POMDP)  in  literature  (Bernstein  et  al., 

2002),  where  at  each  step  when  an  agent  takes  an  action,  a  state  transition  occurs,  and  the 
agent  receives  a  local  observation.  Following  this,  the  environment  generates  a  global 
reward  that  depends  on  the  set  of  actions  all  agents  take.  The  complexity,  however,  of  this 
decentralized  control  model  is  NEXP-hard  (Bernstein  et  al.,  2002),  and  hence  it  is 
computationally  intractable.  In  our  previous  work  (Cheng,  Raja,  &  Lesser,  2012),  we  made 
the  DEC-POMDP  problem  for  a  tornado-tracking  tractable  by  approximating  the  DEC- 
POMDP  with  a  stochastic  DEC-MDP1  model  and  using  a  factored  reward  function  to  define 
a  Nash  Equilibrium  instead  of  the  global  reward  function.  A  necessary  condition  for  stable 
equilibrium  among  agents  in  a  multi-agent  system  is  that  each  agent  plays  a  best-response 
to  the  strategy  of  every  other  agent:  This  is  called  a  Nash  Equilibrium.  We  applied  this 
technique  to  the  MDAP  domain.  We  defined  the  reward  function  of  this  model  to  be 
composed  of  two  different  components:  local  reward  function,  and  global  reward  function. 
The  local  reward  functions  are  dependent  only  on  the  individual  agents’  actions,  while  the 
global  reward  function  depends  on  the  action  of  all  agents. 

The  DEC-MDPmodel  is  defined  as  a  tuple  <S,Jl,7 ,  where 

S  =  x  S2  x . x  Jn  is  a  finite  set  of  factored  world  states,  where  S,-  is  the  state  space  of  agent  j.. 

ctf  =  x  A3  x x  An  is  a  finite  set  of  joint  actions,  where  is  the  action  set  for  agent  j.. 

7:  S  x  Jl  x  5  ->  JS  is  the  transition  function.  T  (s  s.  a)  is  the  probability  of  transiting  to  the  next  state 
after  a  joint  action  g  E  A  is  taken  bv  agents  in  state  s. 

is  a  set  of  factored  reward  functions.  R; :  Sx  A  ->  71  provides  agent  j  with  an 
individual  reward  t;  E  J?i(s.  a)  for  taking  action^ in  state  s. 


We  made  this  a  stochastic  DEC-MDP  by  defining  a  solution  as  a  stochastic  policy  for 
each  agent.  A  stochastic  policy  of  an  agent  is  denoted  by  rii(s)  e  PD  (A),  where  PD  (A),  is 
the  set  of  probability  distributions  over  actions  A.  Stochastic  policies  can  cope  with  the 
uncertainty  of  observation  and  perform  better  than  deterministic  policies  in  a  partial 
observable  environment. 


1  A  DEC-MDP  is  a  DEC-POMDP  with  joint  full  observability  (Bernstein  et  al.,  2002). 
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State  Space 

Feature  1:  Program  ID 

Feature  2:  Current  Year 
Feature  3:  Current  Month 

Feature  4:  Cost  (APB)  Status:  for  9  months,  starting  from  the  current  month 

Feature  5:  Cost  (Contract)  Status:  for  9  months,  starting  from  the  current  month 

Feature  6:  Schedule  (APB)  Status:  for  9  months,  starting  from  the  current  month 

Feature  7:  Schedule  (Contract)  Status:  for  9  months,  starting  from  the  current 
month 

Feature  8:  Performance  (APB)  Status:  for  9  months,  starting  from  the  current 
month 

Feature  9:  Performance  (Contract)  Status:  for  9  months,  starting  from  the  current 
month 

Feature  10:  Funding  (APB)  Status:  for  9  months,  starting  from  the  current  month 

Feature  1 1 :  Funding  (Contract)  Status:  for  9  months,  starting  from  the  current 
month 

Each  of  the  above  features  (4-1 1 )  is  represented  by  one  of  three  colored  bubbles 
(green,  yellow,  and  red)  in  the  Program  Status  page  of  the  DAES  report.  Yellow  bubbles 
refer  to  resolvable  issues,  and  red  bubbles  refer  to  critical  issues.  But  if  there  is  no  issue, 
then  the  feature  is  represented  by  a  green  bubble.  The  number  of  bubbles  starting  from  the 
current  month  indicates  the  number  of  months  during  which  the  issue  will  sustain.  We 
assigned  the  green,  yellow,  and  red  bubble  weights  of  0.0,  0.1,  and  1.0,  respectively. 

Hence,  in  the  feature  value,  the  count  of  yellow  bubbles  appears  at  the  right  side  of  the 
decimal  point,  and  the  count  of  red  bubbles  appears  at  the  left  side  of  the  decimal  point  of 
the  feature  value.  For  example,  consider  the  value  of  Feature  4:  Cost  (APB)  =  4.0.  This 
value  indicates  that  the  cost  (APB)  issue  is  critical  and  that  it  would  continue  to  be  critical  for 
the  next  consecutive  four  months.  Then  it  is  predicted  to  be  resolved. 

Action  Space 

We  capture  both  local  and  non-local  actions. 

Local  action  1  (LAI):  PM  takes  action  to  resolve  APB  cost  issue 

Local  action  2  (LA2):  Contractor  takes  action  to  resolve  Contractor  cost  issue 

Local  action  3  (LA3):  PM  takes  action  to  resolve  APB  schedule  issue 

Local  action  4  (LA4):  Contractor  takes  action  to  resolve  Contractor  funding  issue 

Local  action  5  (LA5):  PM  takes  action  to  resolve  APB/Contractor  funding  issue 

Local  action  6  (LA6):  PM  takes  action  to  resolve  APB  performance  issue 

Local  action  7  (LA7):  Contractor  takes  action  to  resolve  Contractor  performance 
issue 

Local  action  8  (LA8):  PM  initiates  inter-governmental  dialogue  to  resolve  the 
pertaining  issue 
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Non-Local  Action  (NLA):  Coordinate  with  program  /'.  (i  refers  to  a  neighbor 
program) 

Transition  Probabilities 

The  transition  probability  function  will  be  computed  empirically  based  on  the  past 
performance  breaches  of  programs  in  the  network. 

Reward  Function 

The  joint  reward  function  is  composed  of  local  and  global  rewards.  Local  rewards  are 
achieved  both  monthly  and  yearly. 

We  will  calculate  local  reward  value  from  the  Acquisition  Baseline  Program  section  of 
DAES  reports.  The  following  two  parameters  (LR1  and  LR2)  capture  the  change  in  %PAUC 
and  schedule  on  a  monthly  basis.  We  will  use  the  following  code  to  depict  their  changes: 

If  current  %PAUC  <  10%  of  the  APB,  then  status  =  0 

If  current  %PAUC  >  0  &&  PAUC  <  10%  of  the  APB,  then  status  =  + 

If  current  %PAUC  >=  10%  &&  <  15%  of  the  APB,  then  status  =  1 

If  current  %PAUC  >  15%  of  the  APB,  then  status  =  10  (breach  has  occurred) 

Schedule:  #  of  months  beyond  the  threshold. 

LR1:  PAUCMonthly  (APB) 

LR2:  ScheduleMonthly  (APB) 

To  calculate  the  local  reward  value,  which  is  calculated  yearly,  we  will  use  the 
following  parameters  captured  from  SAR  files: 

LR3:  APB  Breach  RDT&E  (Values:  0/1 ) 

LR4:  APB  Breach  Procurement  (Values:  0/1) 

LR5:  APB  Breach  Schedule  (Values:  0/1) 

LR6:  APB  Breach  Performance  (Values:  0/1) 

LR7:  APB  Breach  PAUC  (Values:  0/1 ) 

LR8:  Nunn-McCurdy  Breach  PAUC  (Values:  0/1) 

LR9:  %PAUC  (amount  that  appears  in  SAR) 

For  the  calculation  of  global  reward,  which  is  to  be  calculated  yearly,  we  will  use 
following  parameters: 

GR  1 :  Criticality  value  of  neighbor  1  (CR_ID  of  neighbor) 

GR  2:  Criticality  value  of  neighbor  2  (CR_ID  of  neighbor) 

GR  3:  Criticality  value  of  neighbor  3  (CR_ID  of  neighbor) 

GR  4:  Criticality  value  of  neighbor  4  (CR_ID  of  neighbor) 

Etc. 

Criticality  value  indicates  the  importance  of  the  neighbor  nodes  in  terms  of  creating 
impact  over  the  other  nodes  in  the  MDAP  network.  It  is  based  on  centrality  measures 
defined  in  network  theory  where  the  centrality  quantifies  the  importance  of  the  nodes  in  a 
networked  system  (Newman,  201 1 ).  We  calculated  the  criticality  of  the  nodes  for  the  global 
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reward  based  on  the  composite  parameter  of  a  neighbor  program  from  the  SAR  files  as 
follows: 


CR  =  APB_RDT&E  +  APB_Procurement  +  APB_Schedule  +  APB_Performance 
+APB_PAUC+Nunn-McCardy_PAUC+%PAUC 

Based  on  the  local  and  global  rewards,  we  will  calculate  the  joint  reward  by  summing 
up  the  contributions  of  both  the  local  and  global  reward  functions.  The  aggregate  local  and 
global  reward  value  may  have  different  weights  associated  with  it. 

Understanding  the  Characteristics  of  the  Existing  Data 

In  this  section,  we  describe  the  importance  of  the  data  set  that  facilitates  deeper 
understanding  about  the  dynamics  of  the  MDAP  network.  We  also  enumerate  the  issues 
related  to  the  quality  of  the  data,  as  well  as  about  its  completeness  and  availability  in  the 
subsection  Structure  of  the  Data  and  the  subsection  Availability  of  the  Data.  We  believe  that 
by  addressing  these  issues,  the  accuracy  of  the  proposed  decision-theoretic  model  would 
be  enhanced. 

Significance  of  the  Data  Set 

The  available  data  that  we  used  for  an  in-depth  study  of  the  MDAP_A  funding 
network  offers  significant  insight  into  each  individual  program,  as  well  as  into  their 
interdependency  relationships.  DAES  reports,  which  are  published  monthly,  provide  a 
granular  view  of  the  local  issues  pertaining  to  the  program  and  the  mitigation  actions  that 
have  been  taken  to  resolve  the  issues.  Analyses  of  monthly  forecasting  on  the  program 
features  helped  us  identify  the  root  cause  of  the  program  or  its  absence,  which  in  effect  led 
us  to  search  for  non-local  causes  originated  through  cascading  effects.  SAR  files,  on  the 
other  hand,  provide  a  quantitative  depiction  of  the  program  status  on  the  basis  of  accrued 
breaches,  increase  in  %PAUC,  cost,  and  funding  figures.  This  resource  helped  us  with 
comparative  quantitative  analyses  and  to  gain  insight  into  the  cascading  effects. 

Structure  of  the  Data 

•  We  note  that  none  of  the  performance  reports  directly  capture  the  interdependent 
regions. 

•  Although  the  PE  documents  provide  a  set  of  programs  that  share  a  common 
funding  source,  they  do  not  provide  a  comparative  status  of  the  programs. 

•  The  DAES  reports  show  the  data  interdependency,  but  they  do  not  provide  at 
least  a  summary  status  of  the  data  neighbors. 

•  To  determine  the  cascading  effect  between  MDAP_A  and  MDAP_B,  we  had  to 
build  a  “funding  summary”  table  for  both  the  programs  based  on  base  year 
dollars.  The  existing  SAR  format  provides  only  the  then-year  funding  summary. 
For  comparison  and  analyses,  this  table  should  be  provided  in  terms  of  base 
year  dollar. 

•  We  observe  that  some  DAES  reports  provide  a  better  understanding  of  the 
issues  and  mitigation  measures,  although  others  do  not.  There  should  be  a 
uniform  standard  to  prepare  this  document. 

Availability  of  Data 

We  observe  that  monthly  DAES  reports  for  the  nodes  in  the  MDAP_A  funding 
network  provide  a  very  small  spectrum  of  useful  data  for  analyses.  Although 
some  programs  report  from  2006,  the  complete  data  set  for  all  the  members  of 
the  MDAP_A  network  is  available  only  for  the  years  2008  and  2009.  For  the  year 
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2007,  only  some  programs  possess  a  complete  report.  Some  DAES  reports 
provide  partial  information  (containing  only  the  risk  summary  page);  hence,  they 
are  not  suitable  for  our  analysis. 

•  So  far,  SAR  seems  to  be  the  only  resource  that  captures  some  aspects  of 
interdependency.  But  the  fact  that  SAR  was  not  published  in  2008  caused 
discontinuity  in  our  analyses. 

•  We  find  that  some  programs  stopped  reporting  after  a  certain  time.  Therefore,  we 
had  no  way  to  learn  the  status  of  the  program,  even  if  it  performs  poorly.  This 
unavailability  problem  appears  to  be  a  challenge  understanding  interdependency 
issues. 

Conclusions  and  Future  Work 

We  have  conducted  a  case  study  of  the  MDAP_A  funding  network  based  on  the 
available  DAES  and  SAR  files  for  the  period  2004  to  2010.  In  the  Network  Model  section, 
our  analyses  of  these  disparate  yet  intrinsically  related  data  indicated  that  the  programs  are 
related  to  other  programs  based  on  funding  and  data  relationships.  This  supported  our  belief 
that  a  network-centric  predictive  model  would  be  a  good  candidate  for  MDAP  performance 
analyses.  We  also  noticed  that  although  the  available  data  provides  useful  information  about 
the  MDAPs,  it  was  challenging  to  integrate  and  understand  this  data  coherently  so  that 
network  dependencies  can  be  revealed  accurately. 

In  the  Case  Study  of  MDAP_A  Funding  Network  section,  we  observed  that  issues 
that  led  a  program  towards  experiencing  APB  breach  and/or  increase  in  %PAUC  were  not 
solely  local,  and  that  the  non-local  issues  might  affect  the  performance  of  the  program.  We 
studied  two  related  programs,  MDAP_A  and  MDAP_B,  from  a  local  perspective  based  on 
their  respective  DAES  reports  and  showed  that  local  mitigation  efforts,  although  successful 
at  times,  still  resulted  in  APB  breaches  at  other  times.  Specifically,  we  observed  from  the 
SAR  files  that  the  cost  overrun  of  MDAP_A  in  2009  onwards  might  have  affected  MDAP_B 
in  2010  in  the  form  of  a  procurement  funding  shortfall.  This  observation,  even  if  it  may  not  be 
conclusive,  suggests  cascading  effects  between  neighboring  MDAPs.  Our  study  of  MDAP_B 
in  2009  and  2010  led  us  to  address  two  questions:  (1 )  Why  would  the  procurement  funding 
requirement  increase  in  2009  and  2010?  and  (2)  What  is  the  reason  for  MDAP_B  not 
receiving  its  requested  amount  of  funding  that  resulted  in  a  funding  shortfall  condition  for  two 
consecutive  years?  Although  the  SAR  files  provide  an  answer  to  the  first  question,  which  is 
that  the  increase  in  quantity  led  to  the  need  for  increased  funding,  our  data  did  not  provide 
an  answer  to  the  second  question.  Hence,  it  appears  that  this  lack  of  knowledge  about  one’s 
own  program  domain  (not  being  able  to  understand  the  root  cause  of  the  APB  breach 
issues)  may  result  in  producing  unexpected  cascading  effects  through  the  neighbor 
programs. 

In  A  Decision-Theoretic  Model  for  MDAP  Network  section,  we  first  argued  that  a 
decision-theoretic  model  based  on  MDPs  would  be  a  good  candidate  to  isolating  cascading 
risks  for  the  MDAP  network.  We  then  showed  that  the  partially  observable  state  space  of 
each  program  warrants  a  DEC-POMDP  model,  which  belongs  to  the  class  of  MDPs.  The 
computational  complexity  of  the  original  DEC-POMDP  led  us  to  explore  feasible 
approximations  that  will  still  provide  the  performance  guarantees.  We  are  currently  working 
on  automating  this  decision-theoretic  model  for  the  nodes  in  the  MDAP_A  funding  network. 

We  believe  that  true  joint  capability  relies  on  the  understanding  of  the  scope  and 
challenges  of  the  interdependencies  among  MDAPs.  Our  manual  analyses  of  the  DAES  and 
SAR  documents  for  a  focused  MDAP  case  study  reveal  indications  about  possible 
cascading  effects  and  offer  better  understanding  about  the  root  causes  for  poor 
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performance  of  programs.  In  the  future,  we  plan  to  automate  this  process  based  on  the 
proposed  DEC-MDP  model  leveraging  larger  data  sets.  It  would  be  important  to  observe 
how  the  second  and  higher  order  neighbors  contribute  to  the  cascading  effects.  We  also 
plan  to  extend  these  analyses  for  MDAP  data  network  focusing  on  data  relationships  that 
are  relatively  stable  over  the  multiple  years. 
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Capabilities 


The  future  operating 
environment  will  continue  to  be 
characterized  by  uncertainty, 
complexity,  rapid  change,  and 
persistent  conflict,  => 

DoD  leadership  has  explicitly 
sought  the  capability  to  act 
jointly 


An  integrated  approach  to  strategic 
planning,  capabilities  needs  assessment, 
systems  acquisition,  and  program  and 
budget  development. 


Interdependency 


MDAP 


•  Defined  as  dependence  on  an  external  source  for 

•  data, 

•  money, 

•  staff, 

•  facilities,  or 

•  requirements 

beyond  the  normal  acquisition  workflow. 

•  Data  sources 


SAR 


DAES 


Contractor 

Info 


Docs 


Perspectives 


•  Local  (endogenous) 

•  how  will  my  costs  overrun  this  year  affect  my  performance  next  year? 

•  Non-local  (exogenous) 

•  what  if  my  partner  reneges  on  a  funding  obligation? 

•  how  will  my  cost  overrun  affect  my  neighbors? 
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rogram  Element 
Interdependencies 


Growing  Interdependencies  and  Complexity 
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Cascading  Effect: 
Hidden  Dynamics 
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Specific  Goals 


1 .  Identify  highly  dependent  parts  of  the  MDAP  network. 

2.  Study  effectiveness  of  current  mitigation  forecasting. 

3.  Develop  a  mathematical  model  to  describe  and  predict 
non-linear  cascading  effects  from  one  MDAP  to  another. 

4.  Understand  the  data  collection  process  and  challenges: 

•  missing,  inaccurate  data  etc. 


•  Deterministic/Linear  Methods  (Brown,  Flowe  2010,  2011 
2012) 

-  Use  correlation  to  show  cascading  effects  and  interdependence. 

—  Data:  Entire  network  of  MDAPs  over  several  years. 

-  Top-down  approach;  bird’s  eye  view. 

•  Non-deterministic/Non-linear  methods  (this  work) 

■  What-if  mathematical  models. 

■  Data:  Case-study  of  a  small  set  of  MDAPs  over  several  years. 

■  Bottom-up  approach;  careful  analysis  of  individual  programs  and  their 
interdependencies. 

■  Uncover  early  indicators  of  interdependency  risk  to  isolate  appropriate 
governance  oversight  methods. 


Main 

Contributions 


1 .  Existing  data  features  facilitate  multi-perspective  study. 

2.  Identify  factors  that  cause  mitigation  forecasting  to  falter. 

3.  Non-local  factors  affect  program  outcomes: 

-  “program-centric”  +  “program  network  approach”  for  acquisition  and 
management  is  advantageous. 

4.  Cascading  effects  can  be  recast  as  a  sequential  decision 
problem 

5.  Identify  challenges  inherent  in  the  data  collection  process. 
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Phase  1:  Identify 
"critical"  programs 

[APB  breaches  &  %APAUC] 


^Phase  3:  Study^ 
non-local  reasons 
for  missed 
performance 
estimates 


Three-phase 

Methodology 


Phase  2:  Study  local 
reasons  for  missed 
performance 
estimates 
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2004,  2005 


E  STUDY: 
Evolving  Funding 
Neighborhood 


2009 

None 

None 

None 

None 

None  (2.45K) 

2010 

None 

None 

None 

None  (1.05  'A] 

MDAP  E 


Phase  1 :  “Critical” 


programs 

(SARs  2004-2010) 


Phase  2:  Local 
Factors  MDAP  A 


Contractor's  inability 
to  forecast 

A 

^  s 


Weapons 
procurement  cut 


Phase  2:  Local 
Factors  MDAP  B 


Phase  1  &  2 
Results 


•  MDAP  can  have  more  than  one  type  of  APB  breach  in  a  year 
and  %PAUC  can  still  decrease: 

-  Lag  from  previous  year. 

-  PMs  may  leverage  project  management  triangle  model  (Bethke, 


2003). 


•  Main  cause  for  MDAP_B’s  cost  and  funding  problems  => 
Shortfall  in  requested  funding. 

•  DAES  reports  do  not  provide  obvious  local  (endogenous) 
reasons  for  this  shortfall  in  funding: 

-  For  e.g.  no  new  breaches. 

•  Investigate  the  overlapping  region  between  MDAP_A  and 
MDAP_B  to  identify  possible  non-|r>r'ai  cascading  effects. 
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unding  Summary 
(2004-2010) 


MDAPA 

Baseline 

Quantity 

Current 

Quantity 

%PAUC 

Current  Year 

Re  quire  d  Funding 

(x) 

Re  c  eiv  e  d  Funding 

<y) 

Delta 

(y  -  *) 

2004 

6 

6 

-9.9S 

221.1 

2005 

6 

6 

-11.65 

59S.5 

579. S 

-IS. 7 

2006 

6 

6 

-6.14 

1012.1 

997.3 

-14. S 

2007 

6 

6 

-1.24 

15SS.4 

1574.6 

-13. S 

2009^ — 

6 

3.14 

3163 .2 _ _ _ 

3006.3 

-156.9 

3.S2 

\2 

62.5 

MDAP  A 


Summary  (SB"' 


MDAP  B 


2007  %PAUC  ■ 

2009  Funding  L  MM 

1  Funding 

i,  increase 

yntitv 

Shortfall 

(y-x)  | 

20b^ 

^29574 

0 

4^7^  ^2 

0 

2005 

329574 

32S514 

3.S5 

137.2 

135.5 

-1.7 

2006 

329574 

32S514 

3.S5 

255.5 

250.3 

-5.2 

2007 

329574 

95961 

7.69 

350.5 

34S.1 

-2.4 

2009 

329574 

215961 

-26.92 

644.1 

593.2 

-50.9 

2010 

329574 

22197S 

-19.23 

751.6 

711.1 

^0.5 

MDAP_B 
2010  Cost 
breach 


Table:  MDAF_B  SAR  Funding  Summary  {5 BY)  for  the  period  2004-20 10 
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Decision 
Theoretic  Model 


*  Reasoning  explicitly  about  uncertainty  is  key: 

-  Must  anticipate  various  possible  outcomes  over  time  to  support  effective  decision 
making. 

*  MDPs  provide  a  rigorous  foundation  for  sequential  decision 
making: 

-  Hedging  allows  managers  to  (a)  test  their  decisions  to  avoid  possibility  of  failure  and 
(b)  to  choose  actions  that  ensure  higher  overall  expected  rewards 

-  Computing  optimal  policies  will  support  non-myopic  decisions. 

-  Address  partial-observability  using  a  derivative  called  DEC-MDPs. 

*  Build  Pattern  Knowledge: 

-  Capturing  role  of  interdependencies,  past  performance  and  action  outcomes  across 
MDAPs  in  the  MDP. 


f  State 

FO:  Time  (month) 
FI:  APB  Cost 
F2:  APB  Schedule 
F3:  APB  Perf 
F4:  APB  Funding 
F5:  Info  about 
funding  neighbors 
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Reward 

Breach 


No  Breach. 
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Analysis  Results 

(1/2) 


1.  Contractor  either  underestimates  or  cannot 
accurately  estimate  the  technical  challenges 
and  the  amount  of  required  funding. 


2.  Budget  cuts  ->  delay  schedule  ->  cost 
increases. 


3.  Procurement  funding  shortfall  leads  to  cost 
and  funding  problems. 


Analysis  Results 

(2/2) 


4.  Instances  of  cascading  effects  suggest: 


“program-centric”  +  “program-network”  approach  ->  acquisition 
management  and  oversight 


5.  Recurring  local  issues  =>  either  the  root  cause  is  not 
captured  in  the  DAES  or  the  cause  is  exogenous  to 
the  program  boundary: 

•  PARCA:  Important  ongoing  work  in  root  cause  analysis  is 
encouraging. 

6.  Critical  need  to  design  automated  data  extraction  and 
analysis  methods. 


“  Data 
Characteristics 
and  Needs 


Significance  of  the 

Needs  wrt  Structure 

Availability  of  Data 

Data  set 

of  the  Data 

•  The  available  data  offers 
significant  insight  about  each 
individual  program. 


•  Capture  more  information 
on  interdependencies. 

•  Provide  comparative  status 
of  programs. 

•  Provide  summary  status  of 
the  data  neighbors  in  DAES 
reports. 

•  Uniformity  in  DAES  report 
format  across  programs. 


•  Complete  data  set  for 
MDAP_A  network  is  available 
only  for  the  years  2008  and 
2009. 

•  Existing  programs  sometimes 
stop  reporting  after  operating 
for  a  certain  number  of  years. 


Next  Steps 


•  Study  the  Structural  Properties  of  the  MDAP  Network: 

-  Continue  to  refine  criteria  that  identify  most  “critical”  MDAPs  and  root  cause 
analysis. 

-  Include  PE  docs,  contractor  data  etc. 

-  Study  temporal  behavior  of  cascading  vector  over  time. 

-  Measure  path  length  (n/w  diameter)  and  its  influence  on  cascades. 

-  Determine  cost  of  “jointness”  and  associated  risk  levels. 

•  Data  Extraction  &  Analysis: 

-  Automate  data  extraction  process. 

-  Populate  DEC-MDP  model  automatically  and  compute  distributed  policy. 

•  Data  Needs: 

-  Complete  Data  for  a  subset  of  MDAPs  would  be  very  usefyL-^^  fl 

-  Ideally  take  a  deep  dive  studying  a  few  selected  programs. 
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hase  3:  Non-local 
factors  MDAP  B 


This  observation,  even  if  it  may  not  be  conclusive, 
is  suggestive  of  cascading  effects  between 
neighboring  MDAPs. 


